Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Init plugin #3

Merged
merged 9 commits into from
Mar 15, 2024
Merged

feat: Init plugin #3

merged 9 commits into from
Mar 15, 2024

Conversation

cv65kr
Copy link
Member

@cv65kr cv65kr commented Feb 9, 2024

Reason for This PR

closes: roadrunner-server/roadrunner#1471

Description of Changes

MVP for google pub sub plugin

License Acceptance

By submitting this pull request, I confirm that my contribution is made under
the terms of the MIT license.

PR Checklist

[Author TODO: Meet these criteria.]
[Reviewer TODO: Verify that these criteria are met. Request changes if not]

  • All commits in this PR are signed (git commit -s).
  • The reason for this PR is clearly provided (issue no. or explanation).
  • The description of changes is clear and encompassing.
  • Any required documentation changes (code and docs) are included in this PR.
  • Any user-facing changes are mentioned in CHANGELOG.md.
  • All added/changed functionality is tested.

Summary by CodeRabbit

  • New Features

    • Introduced Dependabot configuration for automatic dependency updates.
    • Added a pull request template to streamline submission descriptions.
    • Implemented GitHub Actions workflows for CodeQL analysis, linting, and testing across environments.
    • Updated .gitignore to better manage project files.
    • Enhanced Google Cloud Pub/Sub plugin with new functionalities for job processing.
    • Added comprehensive tests for job processing and integration with Google Cloud Pub/Sub.
    • Introduced mock implementations and PHP test files for improved testing capabilities.
  • Chores

    • Updated import paths and function signatures in the plugin code to improve integration and performance.
  • Documentation

    • Expanded configuration files for testing and integration setups.

@cv65kr cv65kr requested a review from rustatian February 9, 2024 19:04
Copy link

coderabbitai bot commented Feb 9, 2024

Walkthrough

The repository underwent significant updates to introduce Google Pub/Sub as a jobs driver, enhancing the jobs plugin with cloud-native services support for both AWS and GCP. This includes new configurations, workflows for code analysis and linting, and extensive changes to the pubsubjobs package to manage topics, subscriptions, and message processing efficiently. Additionally, the project saw updates to testing infrastructure, including new PHP test files and mock implementations for robust validation.

Changes

File Pattern Change Summary
.github/... Introduced Dependabot, PR template, CodeQL analysis, Linters, and Google pub sub test workflows.
.gitignore Updated to exclude vendor directories and .DS_Store.
plugin.go Updated import paths and function signatures, added masterPluginName.
pubsubjobs/... Major updates for Google Cloud Pub/Sub integration, including new logic and error handling.
tests/configs/..., tests/helpers/helpers.go, tests/mock/..., tests/php_test_files/jobs/..., tests/jobs_test.go Enhanced testing infrastructure with new configs, helpers, mocks, and PHP test files.

Assessment against linked issues

Objective Addressed Explanation
Add Google Pub Sub as a jobs driver (#1471)

Poem

In fields of code where logic plays,
A rabbit hopped through the GitHub maze.
🐰 With every commit, push, and pull,
It crafted features, wonderful and full.

"Let's connect the clouds," it said with glee,
"AWS, GCP, together, free!"
🌥️ Through lines of code, it danced with joy,
A happy, coding, clever toy.

Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media?

Share

Tips

Chat

There are 3 ways to chat with CodeRabbit:

  • Review comments: Directly reply to a review comment made by CodeRabbit. Example:
    • I pushed a fix in commit <commit_id>.
    • Generate unit-tests for this file.
    • Open a follow-up GitHub issue for this discussion.
  • Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
    • @coderabbitai generate unit tests for this file.
    • @coderabbitai modularize this function.
  • PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
    • @coderabbitai generate interesting stats about this repository and render them as a table.
    • @coderabbitai show all the console.log statements in this repository.
    • @coderabbitai read src/utils.ts and generate unit tests.
    • @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (invoked as PR comments)

  • @coderabbitai pause to pause the reviews on a PR.
  • @coderabbitai resume to resume the paused reviews.
  • @coderabbitai review to trigger a review. This is useful when automatic reviews are disabled for the repository.
  • @coderabbitai resolve resolve all the CodeRabbit review comments.
  • @coderabbitai help to get help.

Additionally, you can add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.

CodeRabbit Configration File (.coderabbit.yaml)

  • You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
  • The JSON schema for the configuration file is available here.
  • If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/coderabbit-overrides.v2.json

CodeRabbit Discord Community

Join our Discord Community to get help, request features, and share feedback.

@github-advanced-security
Copy link

This pull request sets up GitHub code scanning for this repository. Once the scans have completed and the checks have passed, the analysis results for this pull request branch will appear on this overview. Once you merge this pull request, the 'Security' tab will show more code scanning analysis results (for example, for the default branch). Depending on your configuration and choice of analysis tool, future pull requests will be annotated with code scanning analysis results. For more information about GitHub code scanning, check out the documentation.

@rustatian rustatian added the enhancement New feature or request label Feb 10, 2024
@rustatian
Copy link
Member

Hey @cv65kr 👋
Thank you very much for the PR 👍
Are you planning to do something additional on it? Also, I noticed that some secrets are needed to run the workflow. Could you please also comment, where to obtain them?

@rustatian
Copy link
Member

Hey @cv65kr 👋
Sorry for the super long review 😢
Let me merge this PR and take a look at the code a little bit later. So, you won't be waiting for the review like with the uber/tally 😆

@rustatian rustatian marked this pull request as ready for review March 15, 2024 19:37
@rustatian rustatian merged commit 3d38f49 into master Mar 15, 2024
5 of 6 checks passed
@rustatian rustatian deleted the init branch March 15, 2024 19:37
Copy link

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review Status

Actionable comments generated: 21

Configuration used: CodeRabbit UI

Commits Files that changed from the base of the PR and between cfa55cf and d57467a.
Files ignored due to path filters (8)
  • go.mod is excluded by: !**/*.mod
  • go.sum is excluded by: !**/*.sum
  • go.work is excluded by: !**/*.work
  • go.work.sum is excluded by: !**/*.sum
  • tests/env/docker-compose-emulator-local.yaml is excluded by: !**/*.yaml
  • tests/go.mod is excluded by: !**/*.mod
  • tests/go.sum is excluded by: !**/*.sum
  • tests/php_test_files/composer.json is excluded by: !**/*.json
Files selected for processing (22)
  • .github/dependabot.yml (1 hunks)
  • .github/pull_request_template.md (1 hunks)
  • .github/workflows/codeql-analysis.yml (1 hunks)
  • .github/workflows/linters.yml (1 hunks)
  • .github/workflows/linux.yml (1 hunks)
  • .gitignore (1 hunks)
  • plugin.go (3 hunks)
  • pubsubjobs/config.go (2 hunks)
  • pubsubjobs/driver.go (4 hunks)
  • pubsubjobs/item.go (7 hunks)
  • pubsubjobs/listener.go (1 hunks)
  • tests/configs/.rr-declare.yaml (1 hunks)
  • tests/configs/.rr-init.yaml (1 hunks)
  • tests/configs/.rr-jobs-err.yaml (1 hunks)
  • tests/configs/.rr-pq.yaml (1 hunks)
  • tests/helpers/helpers.go (1 hunks)
  • tests/jobs_test.go (1 hunks)
  • tests/mock/logger.go (1 hunks)
  • tests/mock/observer.go (1 hunks)
  • tests/php_test_files/jobs/jobs_err.php (1 hunks)
  • tests/php_test_files/jobs/jobs_ok.php (1 hunks)
  • tests/php_test_files/jobs/jobs_ok_pq.php (1 hunks)
Files skipped from review due to trivial changes (1)
  • .github/pull_request_template.md
Additional comments: 37
.gitignore (2)
  • 15-16: The inclusion of **/vendor/ and vendor/ in the .gitignore file is a good practice to avoid committing third-party code to the repository. This change helps keep the repository clean and focused on source code.
  • 18-18: Adding .DS_Store to .gitignore is a good practice to prevent system-specific files from being committed to the repository, which can clutter the project and is irrelevant to other developers.
tests/php_test_files/jobs/jobs_ok.php (1)
  • 11-11: Creating a Consumer instance without specifying any configuration or connection details directly in the code. Ensure that the Consumer is correctly configured elsewhere, typically in a configuration file or environment variables, to connect to the appropriate Google Pub/Sub topic.
tests/configs/.rr-declare.yaml (1)
  • 11-15: The google-pub-sub configuration specifies a project_id, topic, skip_topic_declaration, and host. Ensure that the host value 127.0.0.1:8085 is appropriate for your testing environment, as it seems to point to a locally hosted instance of Google Pub/Sub (possibly emulated). For production environments, this will need to be updated to point to the actual Google Cloud Pub/Sub service.
tests/php_test_files/jobs/jobs_ok_pq.php (1)
  • 15-15: The sleep(15); call simulates a delay in task processing. While this is useful for testing purposes to mimic real-world processing times, ensure that this delay is appropriate for the tests you intend to run. For performance testing or integration tests, the delay might need adjustment.

Consider making the sleep duration configurable or documented to ensure test flexibility and clarity.

tests/configs/.rr-init.yaml (2)
  • 11-15: The Google Pub/Sub configuration section is well-defined, ensuring that the plugin is correctly configured for testing. However, it's important to verify that the host value (127.0.0.1:8085) matches the expected test environment setup, especially if an emulator is being used for local testing.
Verification successful

The Google Pub/Sub emulator is indeed configured to run on 127.0.0.1:8085, as evidenced by the tests/helpers/helpers.go and tests/env/docker-compose-emulator-local.yaml files. This confirms that the host value in the .rr-init.yaml file matches the expected test environment setup.

Scripts Executed

The following scripts were executed for the analysis:


Script:

#!/bin/bash
# Verify if the emulator for Google Pub/Sub is set up to run on 127.0.0.1:8085
docker ps | grep "google-pub-sub-emulator" | grep "127.0.0.1:8085"

Length of output: 138



Script:

#!/bin/bash
# Search for mentions of "google-pub-sub-emulator" and "8085" within the repository
rg "google-pub-sub-emulator" || echo "No mentions of google-pub-sub-emulator found."
rg "8085" || echo "No mentions of port 8085 found."

Length of output: 437

* 31-44: The pipeline configuration for Google Pub/Sub is correctly set up with two test pipelines (`test-1` and `test-2`). Both are configured to use the `google-pub-sub` driver, which aligns with the PR's objectives. However, it's worth noting that `test-2` does not specify a `prefetch` value, unlike `test-1`. Ensure this omission is intentional and aligns with the testing requirements.
.github/dependabot.yml (1)
  • 1-33: The configuration for Dependabot is correctly set up to update dependencies for Go modules, Docker, and GitHub Actions on a daily basis. Assigning rustatian as both reviewer and assignee ensures that updates are promptly reviewed. However, consider adding more team members as reviewers and assignees to distribute the workload and reduce bottlenecks.
tests/mock/logger.go (1)
  • 17-23: The ZapTestLogger function correctly sets up a mock logger for testing purposes. It's a good practice to isolate logging in tests to prevent cluttering the output and to focus on test-related logs only. This implementation aligns well with the testing strategy.
plugin.go (2)
  • 39-39: The check for the existence of the plugin configuration using both pluginName and masterPluginName is a good practice to ensure that the plugin is only initialized when its configuration is present. This prevents unnecessary initialization and potential errors in environments where the plugin is not used.
  • 61-65: The refactoring of function signatures to use jobs.Queue instead of pq.Queue and the adjustments in DriverFromConfig and DriverFromPipeline functions are in line with the PR's objectives to integrate with the Roadrunner server's job processing capabilities. Ensure that all references to these functions are updated accordingly in the codebase.
.github/workflows/codeql-analysis.yml (1)
  • 1-73: The CodeQL workflow is correctly set up to perform code analysis on push and pull request events targeting the master branch, as well as on a weekly schedule. The configuration specifies Go as the language for analysis, which aligns with the project's technology stack. Ensure that the project's build process is compatible with the Autobuild step or provide custom build commands if necessary.
pubsubjobs/item.go (6)
  • 4-6: Imports strconv, sync, and sync/atomic have been added. Ensure these packages are necessary for the functionality being implemented, particularly if they introduce concurrency or atomic operations which can complicate the codebase.
  • 26-26: The struct field Headers has been renamed to Metadata. This change should be clearly documented in the PR to avoid confusion and ensure that all references to this field throughout the codebase are updated accordingly.
  • 64-66: The addition of the GroupID method. Confirm that this method's implementation aligns with its intended use case and that the Pipeline field is the correct source for a group ID.
  • 73-74: The Headers method has been renamed to align with the Metadata field change. Ensure that this renaming is consistent across the codebase and that all references to the old method name are updated.
  • 150-158: The fromJob function converts a jobs.Message to an Item. Ensure that this conversion correctly handles all fields, especially the new Metadata field, and that it properly initializes the Options struct with the correct values from the job message.
  • 244-250: The stob (string to bool) function has been added. Verify that this function correctly handles the conversion of string values to boolean, especially considering case sensitivity and potential string values that should be interpreted as true or false.
tests/mock/observer.go (6)
  • 1-22: The file header indicates that this is a mock implementation for logging purposes, derived from Uber Technologies, Inc. Ensure that the license and copyright notice are appropriate and that the use of this mock adheres to the terms specified.
  • 31-45: The ContextMap method in the LoggedEntry struct converts the context fields into a map. Verify that this conversion correctly handles all field types and that the map representation is used appropriately in tests or logging.
  • 91-110: Filter methods (FilterLevelExact, FilterMessage, FilterMessageSnippet, etc.) have been added to the ObservedLogs struct. Ensure that these methods correctly filter the logs based on the specified criteria and that they are used appropriately in tests to verify logging behavior.
  • 136-149: The Filter method provides a generic way to filter observed logs based on a provided function. Review this method for correctness and ensure that it is used appropriately in tests to filter logs based on complex criteria.
  • 157-165: The New function creates a new mock logging core and associated observed logs. Ensure that this function is correctly initializing the mock logger and that it is used appropriately in tests to capture and analyze logged entries.
  • 188-195: The Write method in the contextObserver struct is responsible for adding logged entries to the observed logs. Review this method for correctness, especially in terms of how it combines context fields with log fields, and ensure that logged entries are correctly captured in tests.
tests/helpers/helpers.go (7)
  • 29-44: The ResumePipes function is a test helper for resuming specified pipelines. Ensure that this function correctly interacts with the job system via RPC calls and that error handling is appropriately implemented.
  • 47-58: The PushToPipe function is a test helper for pushing jobs to a specified pipeline. Review the job creation process within this function to ensure that it correctly sets up jobs for testing, including the autoAck flag and other job options.
  • 61-82: The PushToPipeDelayed function allows for pushing jobs with a specified delay. Verify that this function correctly calculates and applies the delay to jobs and that it interacts properly with the job system via RPC calls.
  • 100-115: The PausePipelines function is a test helper for pausing specified pipelines. Ensure that this function correctly interacts with the job system to pause pipelines and that error handling is appropriately implemented.
  • 118-140: The DestroyPipelines function is a test helper for destroying specified pipelines. Review the retry logic within this function to ensure that it correctly handles temporary failures and that pipelines are properly destroyed in the job system.
  • 143-164: The Stats function retrieves the state of the job system for specified pipelines. Ensure that this function correctly gathers and returns the expected state information and that it is used appropriately in tests to verify the state of the job system.
  • 167-187: The DeclarePipe function is a test helper for declaring a new pipeline with specified options. Review this function for correctness in setting up pipelines, especially the configuration options, and ensure that it interacts properly with the job system.
pubsubjobs/driver.go (3)
  • 5-6: Imports strconv and strings have been added. Ensure these packages are necessary for the functionality being implemented, particularly if they introduce operations on strings or conversions that could affect performance or correctness.
  • 97-135: > 📝 NOTE

This review was outside the diff hunks and was mapped to the diff hunk with the greatest overlap. Original lines [64-125]

The FromConfig function initializes the driver from configuration. Review this function for correctness in handling configuration values, especially error handling and the initialization of the Google Pub/Sub client. Ensure that the configuration is correctly applied and that any errors are appropriately handled.

  • 152-384: > 📝 NOTE

This review was outside the diff hunks and was mapped to the diff hunk with the greatest overlap. Original lines [131-181]

Similar to FromConfig, the FromPipeline function initializes the driver from a pipeline configuration. Review this function for correctness, especially in terms of error handling and the application of pipeline-specific configurations.

tests/jobs_test.go (3)
  • 299-299: The use of a mock logger (mocklogger.ZapTestLogger) is a good practice for capturing and asserting log output in tests. However, ensure that the logger is adequately capturing all relevant log levels and contexts needed for the assertions later in the test.
  • 366-366: The assertion require.Equal(t, 4, oLogger.FilterMessageSnippet("auto ack is turned on, message acknowledged").Len()) checks for a specific number of log messages. This can make the test brittle if the implementation changes slightly but still behaves correctly. Consider if there's a more robust way to assert the desired behavior without relying on the exact number of log messages.
  • 454-459: The assertions in TestRemovePQ rely heavily on specific log messages. This ties the test's correctness to the logging implementation, which might change over time. If possible, use more direct ways of verifying the system's state or behavior, such as querying the state of the pipeline or the jobs directly.

tests/jobs_test.go Show resolved Hide resolved
tests/jobs_test.go Show resolved Hide resolved
tests/jobs_test.go Show resolved Hide resolved
tests/php_test_files/jobs/jobs_ok.php Show resolved Hide resolved
tests/php_test_files/jobs/jobs_ok_pq.php Show resolved Hide resolved
pubsubjobs/driver.go Show resolved Hide resolved
pubsubjobs/driver.go Show resolved Hide resolved
pubsubjobs/driver.go Show resolved Hide resolved
pubsubjobs/driver.go Show resolved Hide resolved
pubsubjobs/driver.go Show resolved Hide resolved
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
Status: ✅ Done
Development

Successfully merging this pull request may close these issues.

[💡 FEATURE REQUEST]: Google Pub-Sub driver
2 participants